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Abstract —This letter proposes the Faster-than-Nyquist sig¬ 
naling (FTNS) using overlap frequency domain equalization 
(FDE), which compensates the inter-symhol interference (ISI) 
due to hand limiting filters of the FTNS at the transmitter and 
the receiver as well as the frequency selective fading channel. 
Since overlap FDE does not require any guard Interval (GI) 
at the transmitter such as cyclic prefix (CP), higher spectral 
efficiency can he achieved compared to FTNS scheme using 
the conventional FDE. In the proposed method, the equalizer 
weight is derived based on minimum mean square error (MMSE) 
criterion taking the colored noise due to the receiving filter 
into consideration. Moreover, we also give an approximated 
FDE weight in order to reduce the computational complexity. 
The performance of the proposed scheme is demonstrated via 
computer simulations. 

Index Terms —Faster-than-Nyquist signaling. Overlap fre¬ 
quency domain equalization, Inter-symhol interference. Mini¬ 
mum mean-square-error. Colored noise. 


I. Introduction 

ASTER-than-Nyquist signaling (FTNS) HI 121 is con¬ 
sidered as one of key technologies for the fifth gener¬ 
ation (5G) mobile communications systems 13. FTNS can 
increase the transmission rate without increasing the required 
frequency bandwidth by transmitted modulated symbols with 
the rate faster than the Nyquist rate, which ensures inter- 
symhol interference (ISI)-free transmission in band limiting 
channels. It has been analytically shown that the minimum 
Euclidean distance between sequences of binary phase shift 
keying (BPSK) symbols using sine-pulse does not degrade 
even with the rate 25% faster than Nyquist rate. This means 
that, if the maximum likelihood sequence estimation (MLSE) 
is employed at the receiver, transmission rate can be increased 
by 25% without the degradation of the bit error rate (BER) 
performance. However, the MLSE requires high computational 
complexity, especially, when the ISIs due to the transmit¬ 
ting/receiving filters as well as the frequency selective fading 
channel are simultaneously compensated. 

To reduce the complexity, the FTNS receiver using fre¬ 
quency domain equalization (FDE) H has been recently pro¬ 
posed as a sub-optimal method in 0 . Moreover, the idea of 0 
has been extended to the method using frequency domain 
decision-feedback equalizer in 0. Furthermore, the FDE- 
aided soft decision and the multistage serially concatenated 
turbo architecture are applied for the FTNS demodulator in Q. 
However, these conventional FDE-based receivers require the 
insertion of the guard interval (GI) such as cyclic prefix 
(CP) at the transmitter, which results in the reduction of the 
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effective transmission rate. Moreover, the colored noise, which 
is inherent in the received signal of the FTNS, is not explicitly 
considered in 0 and Q, while 0 takes the colored noise 
into consideration but with the limitation that the sampling 
rate at the receiving filter is assumed to be twice the symbol 
rate in the formulation. 

This letter proposes an FTNS transceiver using overlap 
FDE 0191, which has been originally proposed for the FDE 
systems with insufficient CP or without CP. Since the overlap 
FDE does not require the insertion of any CP at the transmitter 
at the cost of slight increase in computational complexity at 
the receiver, the proposed scheme can improve the spectral 
efficiency with the moderate complexity. Also, we adopt the 
FDE weight based on minimum mean square error (MMSE) 
criterion in order to take into account for the impact of the 
colored noise, and derive approximated one-tap FDE weight 
to reduce the computational complexity for the equalization. 
Computer simulation results show that there is a significant 
difference in BER performance between the receivers using 
FDE weights with and without considerations of the impact 
of the colored noise. Moreover, we show that the proposed 
method can achieve better BER performance than that of the 
FTNS transceiver using conventional FDE with CP 0 for a 
fixed transmission rate. In addition, we demonstrate that the 
proposed scheme can achieve 30% higher transmission rate 
than that of Nyquist rate without performance degradation, 
when root raised cosine (RC) filter with the roll off factor of 
0.5 is used for transmitting and receiving filters. 

II. System Model 

Fig[T] shows the configuration of the FTNS transmission 
system model considered in this letter. The transmitted symbol 
s[n], {n G Z) is generated by baseband modulation with 
PSK (Phase Shift Keying) or QAM (Quadrature Amplitude 
Modulation) using the binary sequence from the data source. 
We assume i?{s[n]} = 0 and ii^{s[n]s*[TO]} = o'si5[n — m], 
where E{-} represents the expectation, superscript * a com¬ 
plex conjugate, and (5[n] Kronecker delta. The sequence of 
s[n] is transmitted after passing through the transmitting filter 
h{t) at the interval of T. Here, we assume h{t) to be the 
root Nyquist filter with \h{t)\'^dt = 1, and g(t) = 
f-x h{T)h*{T — t) dr satisfies the Nyquist criterion for the 
symbol period of of Tq. In FTNS, in order to increase the 
transmission rate, T is set to be less than Tq- Thus, we have 
T = 7 ^ 0 , (0 < 7 < 1 ). 

The received signal after matched-filtering with h*{—t) is 
given by 

OO 

r{t) = ^ s[m]q{t-mT) + 'q{t), (1) 
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Fig. I. Proposed FTNS transmission model 


where q{t) is the impulse response of channel including the 
impact of the frequency selective channel 0 for Tmin < 

t < Tmax) transmitting/receiving filters and is given by 

/ OO 

g{T)c{t - T)dT, (2) 

-OO 

and 77 (f) is the noise component given by 

/ OO 

no{T)h*{T - t)dT, (3) 

-OO 

where the white noise no(f) follows the complex-valued Gaus¬ 
sian distribution CAf{0, Nq). The received signal is sampled 
with the sampling period of T, which is the same as the symbol 
period, then the nth sample is given by 

OO 

r[n] = r{nT) = q[n — m]s[m] + 'i][n], (4) 

m— — oo 


where q[n] = q{nT) and ? 7 [n] = r]{nT). Note that the 
mean and the correlation of the sampled noise 77 [n] are given 
by E{r][n]} = 0 and i?{? 7 [n]? 7 *[rn]} = Nog{(n — m)T), 
respectively. 


III. Proposed FTNS Receiver with Overlap FDE 

In the proposed method, overlap FDE [|8l,||9l,||II] is em¬ 
ployed to compensate ISIs caused by band limiting filters (i.e., 
transmitting and receiving filters) as well as the frequency 
selective channel. Since overlap EDE performs equalization 
with block-wise signal processing at the receiver, while the 
transmitted signal need not be block-wise, we firstly give a 
block representation of the received signal. Assuming that 
g{t) = 0 for |f| > vT, z/ S Z as in Q by truncating tails of 
the impulse response, a received signal block of size N defined 
as r[k] := {r[k] ■ ■ ■ r[k + N — I])"'", where the superscript T 
represents the transpose, is given by 

r[k] = Qos[k] + Qis[k + N] + r][k], (5) 


where 




Qo — 


g[llmin] 



\ 






V 



^[llmax]/ 

(6) 


and 


/ 


Qi — 


^^min 







(7) 


\ dnmax 1 ... Q^min j 

Also, Ttmin — ^ “t” r7^inin/7^1 and Ttmax = ZZ -p [Ti^ax/T^J. 

where [a:] and [a:J are the smallest integer not less than x and 
the largest integer not greater than x, respectively. In addition, 
s[k] := (s[fc - ^max] ‘ ‘ s\k ^max -I- — 1])'^ and r][k] := 

{r][k] ■■■g[k + N - 1])"'". 

By defining Q := Qq -f Qi, (|5]| can be rewritten as 


r[k] = Qs[fc] -I- Qi{s[k -f TV] - s[fc]) -f T][k]. ( 8 ) 


It should be noted here that Q is a circulant matrix, which 
can be diagonalized by using A^-point DET (discrete Eourier 
transform) matrix D, whose {n,m) {n,m = 
element is as Q = D^AD, where 

the superscript H denotes Hermitian transpose and A G 
is a diagonal matrix whose diagonal elements are the DET of 
the first column of Q. Thus, if we apply EDE for Q ignoring 
the second term of right hand side of (O, which represents 
inter-block interference and ISI components due to the lack of 
CP, the EDE output is given by 

D^WDr[k] = D^WADs[k] 

+ D^WDQi{s[k -f A] - s[k]) + D^WDr][k], (9) 

where W G is an EDE weight matrix. 

Since it is empirically known that head and tail of the signal 
block at the EDE output suffer from the residual interference 
due to the second term of ®, only the middle part of 
D^WDr[k] is used as the output of overlap EDE. Specif¬ 
ically, discarding p elements from the head and q elements 
from the tail, the output of the overlap EDE is given by 

f[k] = {OMxp,lM,OMxq)D^WDr[k], (10) 


where M = N — p — q is the size of the overlap EDE output, 
Im represents the identity matrix of size M x M and Omxp 
a zero matrix of size M x p. The next block processing is 
performed for the received signal block of r[k + M] = {r[k + 
M] ■ ■ • 7 '[fc + M + N — 1])"'" in the same way as described 
above. 

Then, we consider the EDE weight based on MMSE cri¬ 
terion. If we ignore the second interference term in ®, the 
MMSE-EDE weight can be obtained by solving the following 
optimization problem: 

Wc = arg min ^ E\^\\D^WD{Qs[k] + r][k]) — s[fc]|p}. 

(11) 

The MMSE-EDE weight is obtained as 

Wc = (^AA“ + ^DE{r][k]ri^[k]}D^'^ . (12) 

Note that since DE{T]ri[k]^[k]}D^ is not diagonal due to 
the colored noise, Wc is also non-diagonal. Thus, MMSE- 
EDE is not achieved with one-tap EDE, which may spoil the 
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Fig. 2. BER performance of FTNS using FDE with CP for different weights 


motivation to utilize FDE because of the high computational 
complexity. 

In order to reduce the complexity, the proposed FTNS 
scheme obtains the FDE weights by approximating 
DE{riri[k^[k]}D^ with a diagonal matrix, whose diagonal 
entries are the same as those of DE{rirj^\D^. Then, the 
approximated FDE weight of Wc can be given by 

W, = AH , (13) 

where = diag($^[ 0 ],— 1 ]) is a diagonal matrix 
composed by the power spectrum density of the noise 

_ AT-liV-l 

^ E E 5(a - (14) 

m—0 

Note that, if we solve (fTTT l with the assumption of white 
noise, we obtain FDE weight of 

Ww = A^ (^AA^ + ^Ijv^ , (15) 

which is widely employed in conventional FDE systems 
including 0 . 

IV. Simulation results 

We evaluate the BER performance via computer simulations 
in order to conhrm the effectiveness of the proposed method. 
In all the simulations, binary PSK is employed for baseband 
modulation, and root RC filter with roll off factor 0.5 is 
used for transmitting and receiving filters. Thus, the Nyquist 
rate corresponds to 0.67 [bps/Hz]. For FTNS, transmission 
rates are set to be 10% (0.73 [bps/Hz]), 20% (0.80 [bps/Hz]), 
30% (0.87 [bps/Hz]) and 40% (0.93 [bps/Hz]) higher than the 
Nyquist rate. We evaluate the averaged BER by counting error 
using symbol by symbol detection after compensating ISI. The 



Fig. 3. RMSE between FDE output and transmitted symbol for each symbol 
position in a block 


DFT size of FDE is set to be 512 and, in the block processing 
at the receiver, g(t) is truncated with ly = 10 . 

In order to confirm the impact of the colored noise, we 
firstly compare the BER performance of FTNS transmissions 
in AWGN (Additive White Gaussian Noise) channel using 
FDE with CP of length 20 symbols for different FDE weights 
in Sect.III, i.e., (a)Ww, (b)Wc, and (c)Tf7;. Fig. |2] shows 
the BER performance against E},/Nq. Here, 7 is set to be 
7 = {0.875, 0.802, 0.740, 0.687}, which respectively corre¬ 
spond to rates of (0.73, 0.80, 0.87, 0.93} [bps/Hz], where 
the rate loss due to CP is taken into consideration. From Fig. |2] 
we observe the BER performance of (b) is significantly better 
than that of (a) for the same transmission rate especially for 
smaller 7, which implies the importance to consider the impact 
of the colored noise. Also, we can observe that (c) can achieve 
comparable performance to (b) for the same transmission rate, 
which demonstrates the validity of the approximation in the 
derivation of (c). 

Then, we demonstrate the validity of the proposed approach 
using overlap FDE for the FTNS transmission. Fig. 0 shows 
the root MSE (RMSE) between the FDE output using the 
weight (c) and the actual transmitted symbol averaged over 
1,000 FDE outputs versus the symbol position in a signal 
block, where the additive noise power Nq is set to be 0 and 
the channel is flat with a fixed gain of 1. Here, 7 is set to 
be (0.909, 0.833, 0.765, 0.714}, which correspond to rates 
of (0.73, 0.80, 0.87, 0.93} [bps/Hz], respectively. From the 
results, we can see that RMSE values at the middle part of 
the block are significantly smaller than the head and the tail 
of the block for all the values of 7, and that we only have to 
remove rather small portions of the FDE output, which means 
the increase in computational complexity at the receiver due 
to the employment of overlap FDE is rather limited. 

Figs, m and |5] show the BER performance of the proposed 
FTNS scheme using overlap FDE in AWGN channel and 
10-tap frequency selective Rayleigh fading channel having 
maximum duration of 16T, respectively, for different values of 
7 used in Fig. [3 Also, the performance of FTNS using FDE 
with CP 0 is also plotted in the same figures for a comparison 
purpose. In both schemes, the FDE weight of (c) is employed 
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Fig. 4. BER performance compaiison between the proposed FTNS scheme 
and FDE with CP O ((I): AWGN channel) 
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Fig. 5. BER performance comparison between the proposed FTNS scheme 
and FDE with CP [5] ((11): lOtap Rayleigh fading channel) 


to cope with the colored noise, while (a) is used in the original 
scheme in Q. In the proposed FTNS scheme, p and q are set 
to be p = <7 = 30 for the AWGN channel and p = q = 128 
for the frequency selective fading channel. In the scheme using 
FDE with CP, the length of CP is set to be 20 for the AWGN 
channel and 36 for the frequency selective fading channel, and 
7 is set to be {0.875, 0.802, 0.740, 0.687} for the AWGN 
channel and {0.849, 0.779, 0.719, 0.667} for the frequency 
selective fading channel, which results in the same effective 
transmission rates as those of the proposed FTNS scheme. 
From the results in Figs. |4] and |5] we can observe that the 
proposed FTNS scheme can achieve better BER performance 
than that of the scheme in El for the same rate. Moreover, 
in both figures, the BER performance of the proposed ETNS 
scheme with the rate less than or equal to 0.87 [bps/Hz] is 
comparable to that of the Nyquist rate transmission, which 
means that 30% higher transmission rate than the Nyquist rate 
transmission can be achieved with the proposed ETNS scheme 
without BER performance degradation if root RC filter with 
roll off factor 0.5 is employed for transmitting and receiving 
filters. 

V. Conclusion 

In this letter, we have proposed the ISI compensation 
method using overlap EDE for ETNS transmission. In order 
to take account for the impact of colored noises due to 
receiving filter, we have proposed an approximated MMSE 
weight for the one-tap EDE. Computer simulation results show 
that the colored noise should be explicitly considered in the 
FTNS using FDE. Moreover, we have confirmed that, with the 
proposed scheme, 30% higher rate than the Nyquist rate can 
be realized without BER degradation root RC hlter having roll 
off factor 0.5 is used as band limiting filters. 
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